Ac 2007-2977: Use of a Neural Network Model and Noncognitive Measures to Predict Student Matriculation in Engineering

Engineering students’ affective self-beliefs prior to their first year have the potential to help researchers better understand various issues related to student retention and engagement. This paper examines whether a neural network model based on student noncognitive characteristics can be used to predict student persistence in engineering, and the influence of gender in the predictive model. Eight noncognitive measures (i.e., academic self-efficacy, academic motivation, leadership, metacognition, career, type of learner (e.g., deep vs. surface), teamwork, and expectancy-value) serve as independent parameters to an artificial neural network (NN) that is used to predict student persistence within engineering school at the end of first year. A feed-forward neural network model with back-propagation training was developed to predict third semester retention of a cohort of first-year engineering students (N=1,523) at a large Midwestern university. The model constituted of 159 primary nodes corresponding to 8 noncognitive factors described by a 159 item instrument. The resulting model was shown to have a predicative accuracy of 82% for retained students after their first year and 30% for non-retained students. Significantly decreasing the number of inputs (i.e., only using those items that appeared to have the strongest influence) had little impact on the predicative accuracy of the retained students. However, the reduction in inputs decreased the predictive accuracy of the non-retained students by approximately 10%. Results for the same cohort also indicate that the neural network prediction rate is independent of gender. Introduction Engineering programs typically attract the top graduates from high school in terms of grade point average (GPA) and standardized test scores, but attrition out of engineering continues to be a major issue; programs often see some of the most statistically qualified students leave engineering for other majors or drop out of college altogether. In 1975, attrition in the freshman year in engineering was about 12%, increasing to about 25% by 1990 (Beaufait, 1991). In a large study of over 25,000 students at over 300 universities, Astin (1993) found that only 47% of students who begin in engineering graduate with an engineering degree. The National Academies’ report “Rising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future” reports that undergraduate programs in science and engineering have the lowest retention rate among all academic disciplines. The National Academies describes the importance of advances in engineering and technology as crucial to the social and economic conditions for the United States to compete, prosper, and be secure in the global community in the 21st century (Augustine, 2005). One common misconception is that students leave engineering due to lack of academic ability. Studies have shown little difference between the academic credentials of students who remain in engineering and those who leave (Besterfield-Sacre et. al.,1997, Seymour 1997). While there is a positive correlation between GPA and retention, GPA alone doesn’t predict student attrition. Studies have shown that models incorporating cognitive variables such as student high school math and science success (Jagacinski, 1981), strong interest in science (Astin 1992), and higher confidence in basic engineering knowledge and skills (Besterfield-Sacre et. al.,1997) are able to establish a correlation between cognitive variables and retention, but these variables are clearly not single factors in a model to predict retention. Instead, a model using both cognitive and noncognitive – or affective – characteristics shows the greatest promise to accurately identify students who may leave engineering or who may benefit from interventions (Astin et. al.,1992, Felder 1993, Besterfield-Sacre et. al., 1997). In a 2002 study to investigate the predictive relationship between six variables (high school GPA, SAT math score, SAT verbal score, gender, ethnicity, citizenship status) and retention and graduation, Zhang (2002) found that high school GPA and SAT math scores were the best predictor of retention and graduation, while SAT verbal was inversely related. Gender, citizenship and ethnicity were sometimes found to be predictors, but this varied from campus to campus. Astin (et. al., 1992) found that the student’s record in high school was the best predictor of academic success, and performance on standardized tests also had a positive correlation. Zhang (et. al., 2002) identified self-efficacy and physical fitness as positive predictors of freshman retention in a study of several cognitive and affective characteristics. These studies were valuable in identifying characteristics that were predictors for retention, but did not address multiple factors and their interaction as predictors. Instruments designed to assess freshman success include the Pittsburgh Freshman Engineering Attitudes Survey (PFEAS), consisting of 50 items relating to 13 student attitude and self-assessment measures, used to measure differences in student attitudes before and after the freshman year (Besterfield-Sacre, et. al. 1997, Besterfield-Sacre, 1999) The Cooperative Institutional Research Program (CIRP) Freshman Survey covers a wide variety of attributes from financial considerations to attitudes toward school to high school academic performance. A study of academic background variables and the CIRP showed that academic background variables were predictors of future grade performance, but no correlation to retention was reported (House, 2000). These and similar studies indicate that student attitudes and other noncognitive characteristics must be incorporated into any model used to predict retention. These studies and the existing issues with attrition within engineering lead to the question: if no one characteristic has been shown to sufficiently predict student attrition, can a model be developed to take multiple factors and their interaction into account? Data on a set of eight noncognitive variables was collected and analyzed (Maller, 2005, Immekus, 2005). A neural network model incorporating these noncognitive variables allows the investigation into not only the predictive nature of these characteristics, but the predictive possibilities of their interaction in attrition within engineering. Data collection and Instrumentation The sample in this study included 1,523 incoming first-year engineering students (292 females, 1,231 males) at a large Midwestern university during the 2004-2005 academic year. Ethnicity was as follows: 2.05% African American, 0.51% American Native, 10.18% Asian/Pacific Islander, 2.64% Hispanic, 82.43% Caucasian, 2.20% Other. The students’ non-cognitive measures were collected across eight scales (completed prior to the freshman year): Leadership (20 items), Deep vs. Surface Learning (20 items), Teamwork (10 items), Self-efficacy (10 items), Motivation (25 items), Meta-cognition (20 items), Expectancy-value (26 items), and Career Indecision (28 items). All Cronbach’s coefficient alphas for these eight scales were ≥ .80, except for the Teamwork scale (r=.74). Multiple studies have supported the scales’ construct validity based on the results of confirmatory factor analyses (Immekus et. al., 2005, Maller et. al., 2005). Inside these major scales, there were subscales consisting with various numbers of items; for example, under the measure academic motivation, there are four subscales: control, curiosity, career and challenge. Students’ persistence statuses were collected at the beginning of every semester following their freshman year. The investigation in this study focuses on the persistence status at beginning of third semester right after the freshman year. Research Methods Artificial Neural Networks (ANNs): A typical neural network model is an information processing system consisting of inputs, interconnected neurons or nodes as processing units, and output layers. New neural networks must be trained with existing data so it can learn from the examples. During the training process, the weights associated with the links between neurons are adjusted by learning or adapting through the data in repetition. Properly trained neural networks have been widely used in various prediction applications in many areas of engineering. In this study, a feed-forward neural network with back-propagation training algorithm was used to develop models for predicting freshman engineering students’ persistence in engineering. The activation function utilized in these models is log-sigmoid function (Demuth, 1998). All neural network models in this study were developed using Matlab version R2006b from Math Works Inc. The input data used in the training and testing processes are the noncognitive survey items and gender information collected from 1523 freshman engineering students during 2004-2005, as described in previous section. The dependent variable, persistence, is defined in terms of a student’s enrollment status at the start of his/her the third semester. Classification for the status of students’ persistence In this study, the status of students’ persistence after their first year in engineering was classified into five possible categories as described in Table 1. Students who are ‘retained’ in engineering fall into the first two groups: lower-division and upper-division engineering. Students who are ‘not retained’ are those who have transferred or left the university. Engineering freshmen students’ status after 1st year Possible statuses Dichotomous statuses Upper-division engineering: completed first year requirements and move to upper divisions (UE) Lower-division engineering: still remained in the first year program (LE) 1. Retained in engineering Transferred to Science or Technology schools in the same university (ST) Transferred to schools in the same university other than Engineering, Science or Technology (O) Left the university (L) 2. Not retained in engineering Table 1. Classification for the status of students after the first year Prediction performance measures for dichotomous prediction: The performance measures considered in this study are: 1) overall prediction accuracy, 2) sensitivity, 3) specificity, 4) accuracy for “not retained” prediction, and 5) accuracy for “retained” prediction (Larpkiataworn, 2003). Result of prediction Actual Persistence Status Not retained Retained Not Retained True (A) False (B) Retained False (C) True (D) Table 2. Example classification table. Note: A, B, C, D represent the numbers of observations within each classification. The overall prediction accuracy measures the fraction of accurate predictions within the total number of all observations. Its range is 0 to 1, and perfect score is 1, which corresponds to 100% prediction accuracy. Overall prediction accuracy, is defined as: Overall prediction accuracy = A D A B C D +